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and  on  the  Web.  We  compared  the  performance  of  the  students  in  classroom  to  those 
taking  the  course  online.  Specific  design  principals  were  used  for  the  Web  site  in  order  to 
achieve  the  best  interface  to  deliver  the  course  material. 
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I.  INTRODUCTION  ON  DISTANCE  EDUCATION 


A,  THESIS  ORGANIZATION 

Before  we  get  into  the  details  of  Web-based  Training,  we  would  like  to  give  the 
reader  an  overview  of  this  work’s  structure.  In  the  introduction  we  summarized  the 
questions  that  we  answered  and  the  motivation  behind  our  research.  We  explained  ‘What 
is  distance  learning’  and  why  it  became  such  an  interesting  issue  for  the  education 
community. 

Then,  in  Chapter  II,  we  listed  the  different  forms  of  distance  learning  and  focus  on 
the  use  of  the  Web  for  delivering  course  material  to  students.  Guidelines  for  evaluation  of 
Web-based  courses  were  included  in  Chapter  II.  In  Chapter  III,  we  summarized  the 
theoretical  designed  principles  of  an  effective  Web-based  course,  both  for  interface  and 
content.  In  Chapter  IV,  we  explain  how: 

•  We  implemented  the  design  principles  mentioned  in  Chapter  III  to  our 
Web  site. 

•  We  formed  two  classes  of  students,  one  online  and  one  in  classroom  in 
order  to  compare  their  performance. 

•  We  overcome  difficulties  developing  the  course 

•  We  interpreted  the  research  results 

Finally,  in  Chapter  V  we  presented  the  conclusions  from  our  research,  and  posed  some 
questions  for  future  research. 

B,  OUR  RESEARCH  QUESTIONS 

The  main  concern  was  to  evaluate  and  to  compare  the  performance  of  students 
who  take  a  course  online  to  the  performance  of  students  who  take  the  same  course  in  the 
classroom.  Then,  we  wanted  to  identify  the  difficulties  that  arise  when  someone  applies 
instructional  and  usability  design  principles  to  distance  education  Web  sites.  Thirdly,  we 
studied  the  response  of  students  to  learning  online.  Fourth,  we  measured  the  time  we 
needed  to  develop  such  a  course  and  compared  it  to  traditional  courses.  Finally,  we 
gained  experience  on  how  to  develop  online  courses. 

In  order  to  answer  those  questions,  we  passed  through  several  steps: 

1 


•  Step  one:  Seareh  for  design  principles  and  known  difficulties  of  Web  design. 

•  Step  two:  Select  the  type  of  course,  collect  the  course  material  and  define  the 

participating  student’s  groups. 

•  Step  three:  Design  the  site  from  scratch  and  run  usability  tests. 

•  Step  four:  Select  the  platform  to  host  our  site  and  study  the  software  program  to 
develop  the  site  with. 

•  Step  five:  Code  the  Web  site  and  publish  it. 

•  Step  six:  Offer  the  course  in  the  classroom  and  online  in  parallel. 

•  Step  seven:  Evaluate  the  course  using  quizzes  and  online  surveys. 

•  Step  eight:  Interpret  the  results  of  our  survey. 

C.  OUR  MOTIVATION 

The  motivation  behind  this  research  was  the  intention  of  the  Hellenic  Navy  to 
develop  organized  distance  learning  courses  for  its  personnel.  The  Navy  is  specifically 
interested  in  online  courses  that  allow  military  instructors  to  develop  courses  but  not 
interrupt  their  operational  duties  and  also  in  giving  the  opportunity  to  a  broader  number 
of  active  military  personnel  to  attend  courses  and  seminars  no  matter  where  they  are 
stationed. 

D,  WHAT  IS  DISTANCE  LEARNING  EDUCATION? 

This  is  a  natural  question  that  is  easier  to  ask  than  answer.  This  is  true  for  the 
question  “What  is  distance  education”  for  several  reasons.  First,  distance  learning  has 
multiple  meanings.  Distance  can  mean  geographical  distance,  time  distance,  and  possibly 
even  intellectual  distance.  Second,  the  term  ‘distance  education’  has  been  applied  to  a 
tremendous  amount  of  programs  serving  numerous  audiences  via  a  wide  variety  of 
media.  Some  use  print,  some  use  telecommunications,  and  many  use  both.  Finally,  rapid 
changes  in  technology  challenge  the  traditional  ways  in  which  distance  education  is 
defined.  Dan  Coldenway  of  Canada’s  Athabasca’s  University,  provided  a  framework 
useful  in  helping  to  define  four  ways  in  which  education  can  be  practiced,  [ref  1  p.7]  This 
framework,  which  considers  the  two  variables  of  time  and  place,  gives  insight  into 
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different  approaehes  to  the  praetiee  of  education  and  distance  education.  Combinations 
of  time  and  place,  result  in  four  approaches  to  education.  The  four  are  (1)  same  time, 
same  place,  (2)  different  time,  same  place,  (3)  same  time,  different  place,  and  (4) 
different  time,  different  place  education. 

Traditional  education  takes  place  at  the  same  time  in  the  same  place.  This  is 
typically  the  regular  self-contained  classroom  that  most  often  is  teacher  centered. 
Different  time,  same  place  education  means  that  individual  learning  occurs  in  a  learning 
center  or  that  multiple  sections  of  the  same  class  are  offered  so  students  can  attend  the 
class  in  the  same  place  at  the  time  they  chose.  This  is  education  that  is  available  at 
different  times  to  students  but  in  the  same  place,  such  as  the  media  center  or  computer 
laboratory. 

The  last  two  categories  focus  on  education  occurring  in  different  places. 
Instruction  can  take  place  in  different  places  at  the  same  time  when  telecommunication 
systems  are  used.  Often,  television  is  used  to  connect  the  local  classroom  with  the 
teacher  and  the  students  to  learn  from  a  distance.  Satellite,  compressed  video,  and  fiber¬ 
optic  systems  are  increasingly  used  for  same  time,  different  place  education.  This 
approach  is  called  synchronous  distance  learning. 

Students  can  also  learn  at  different  times  and  in  different  places.  Coldeway  has 
said  that  the  purest  form  of  distance  education  occurs  at  different  times  and  in  different 
places  [ref  1  p.8].  In  other  words,  students  chose  when  and  where  to  learn  and  when  and 
where  to  access  instructional  materials.  Recently,  World  Wide  Web  courses  have  been 
offered  to  students  anywhere  they  have  access  and  whenever  they  choose.  This  approach 
is  called  asynchronous  distance  learning. 


E,  DISTANCE  EDUCATION  TODAY 

In  the  last  few  years,  distance  education  has  become  a  major  topic  of  education. 
In  1998,  there  were  over  50  conferences  dealing  with  some  aspect  of  distance  education 
and  almost  every  professional  organization’s  publications  and  conferences  have  shown  a 
huge  increase  in  the  number  of  distance  education  related  articles  and  papers.  [Zvacek 
2000]. 
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Many  educators  are  making  grand  claims  how  distance  education  is  likely  to 
change  education  and  training.  Certainly,  the  concept  of  distance  education  is  exciting 
and  recent  hardware  and  software  innovations  are  making  telecommunications  distance 
education  systems  more  available,  easier  to  use  and  less  costly.  Distance  education  has 
begun  to  enter  the  mainstream. 

However,  distance  educators  faced  two  fronts.  First,  students  say  they  do  not 
really  want  to  learn  from  a  distance.  They  prefer  meeting  and  interacting  with  the 
learning  group  and  the  instructor  in  the  classroom,  seminar  room,  or  the  laboratory. 
Students  report  that  they  value  the  presence  of  learning  groups,  that  the  informal 
interactions  that  occur  before  and  after  class  are  valuable  components  of  their  learning 
experience.  Second,  there  is  evidence  that  students  are  increasingly  demanding  to  be  able 
to  learn  from  a  distance.  They  want  to  be  able  to  supplement,  and  even  place  or  replace, 
conventional  learning  experiences  with  distance  education  programs.  Students  say  this  is 
because  there  are  many  other  course  considerations  besides  personal  preferences  that 
motivate  them,  especially  whether  and  when  they  learn. 

These  opposing  preferences  pose  a  dilemma  for  the  educational  community. 
Should  resources  be  dedicated  to  improving  traditional  education  with  different  structural 
buildings,  classrooms,  and  offices,  should  students  be  transported  to  these  facilities?  Or 
should  funds  be  used  to  develop  modern  sophisticated  telecommunication  systems?  The 
trend  seems  to  be  towards  communication. 

Because  of  advances  in  technology,  effective  educational  experiences  can  be 
provided  for  students,  no  matter  where  they  are  located.  In  other  words,  technologies  are 
now  available  to  develop  cost-effective  distance  learning  systems.  The  practice  of 
distance  education  has  dramatically  changed  in  the  last  decade.  Traditional  approaches  to 
distance  education  based  on  delivery  and  broadcast  media  technologies  are  no  longer  as 
relevant  to  the  field  in  the  United  States  as  they  once  were. 

Distance  education  is  now  often  defined  as  institution  based,  formal  education 
where  the  learning  group  is  separated  geographically,  and  when  interactive  data 
communication  systems  are  used  to  connect  students,  resources,  and  instructors. 
[Albright,  2000] 
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II.  USING  THE  WEB  FOR  TRAINING. 


A,  WHY  WEB  BASED  EDUCATION? 

1,  Is  the  Web  the  right  means? 

Traditional  universities  face  some  basic  questions  when  confronted  with  this  new 
electronic  environment.  Can  the  Internet  be  used  effectively  for  their  educational 
purposes?  Can  Web-based  teaching  provide  students  with  the  same  (or  perhaps  better) 
quality  of  learning  as  the  traditional  face-to-face  environment?  Should  traditional 
universities  go  into  Web-based  teaching  and  compete  with  virtual  or  non-profit 
universities? 

Well-established  universities  are  re-examining  their  mission  and  looking  for 
different  or  supplemental  ways  of  accomplishing  them  (Berge,  1999;  Laurillard,  1993; 
and  Nasseh,  1998;)  Such  re-defmitions  of  non-profit  universities  often  includes  the  goal 
of  providing  life-long  quality  learning  to  as  many  students  as  possible  without  limitations 
of  time,  place,  language,  and  individual  economic  status.  In  order  to  achieve  this  goal, 
universities  are  moving  from  their  traditional  emphasis  on  classroom  instruction  to  an 
environment  where  learning  can  be  pursued  through  any  media  of  instruction,  be  it  the 
classroom,  television,  or  the  Web. 

Advances  in  information  technology  and  telecommunications  are  allowing  Web- 
based  courses  to  replicate  more  seamlessly  the  features  of  face-to-face  instruction 
through  the  use  of  audio,  video,  and  high-speed  Internet  connections  that  facilitate 
synchronous  and  asynchronous  communication  in  chat  groups,  Web  discussion  boards 
and  virtual  forums.  Traditional  instructional  activities,  such  as  lecture  preparation, 
delivery,  student  participation,  discussion,  feedback,  and  evaluation  can  easily  be 
translated  to  the  Web  environment.  With  the  ever-growing  demand  for  technologies  that 
allow  virtual  classrooms  to  more  fully  replicate  all  features  of  face-to-face  environments, 
market-pull  forces,  rather  than  a  technology  push-model  are  driving  innovation. 

Some  of  the  strongest  pressure  for  changes  in  higher  education  is  coming  from 
students.  A  growing  segment  of  working,  self-motivated  students,  want  to  acquire  skills 
they  feel  are  useful  and  also  want  to  be  able  to  choose  how  they  will  learn  those  skills. 
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Convenience  is  becoming  increasingly  important  and  there  is  a  growing  demand  for 
education  that  is  [Berge  1998]; 

•  Independent  of  time  and  space 

•  Oriented  toward  goals  and  outcomes 

•  Centered  in  the  student/  learner 

•  Geared  to  active,  hands-on  learning  and 

•  Able  to  accommodate  differences  in  skills  and  language 

2,  The  Learning  Environment  and  the  Weh 

The  learning  environment  has  two  critical  dimensions;  time  and  place.  We  can 
classify  this  teaching  environment  into  four  major  types: 

•  Type  1.  Same  time  (synchronous),  same  place  in  a  traditional  classroom.  It 
represents  the  traditional  face-to-face  classroom  where  students 
congregate  at  the  same  time  in  the  same  place,  to  be  taught  simultaneously 
the  same  material  by  the  same  person. 

•  Type  II.  Any  time  (asynchronous),  same  place  represents  environment 
where  students  come  at  different  times  to  receive  instructions  at  the  same 
place,  such  as  a  laboratory,  library  or  an  information  center. 

•  Type  III.  Same  time  (synchronous),  any  place  means  distance  learning, 
where  students  are  from  widely  dispersed  geographic  areas  and  can  be 
taught  simultaneously  through  one-way  or  inter-active  audio  and  video 
technology.  . 

•  Type  IV.  Any  time  (asynchronous),  any  place  has  traditionally  been 
represented  by  correspondence  courses. 

The  Web  can  be  used  to  support  or  simulate  all  four  types  of  teaching 
environments.  When  synchronous  teaching  environments  (Type  I  and  III)  are  enriched 
with  live  Internet  connections  and  projection  capabilities,  the  Web  can  be  used  to  support 
or  simulate  lectures,  case  discussions,  and  classroom  interactions  in  multiple  ways  by: 
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•  Serving  as  a  platform  for  simultaneously  delivering  presentations  (text, 
audio,  and  video)  to  students  in  a  classroom  (Type  I),  or  dispersed 
throughout  the  world  (Type  III). 

•  Allowing  synchronous  virtual  visits  to  sites  dedicated  to  relevant  topics  or 
organizations. 

•  Enabling  real-time  or  almost  synchronous  discussions  through  text-based 
technologies  such  as  chat  rooms  and  Web  boards. 

•  When  the  Web  is  used  in  Type  II  environments,  students  gain  access  to  an 
unprecedented  wealth  of  multimedia  information,  tutorials,  materials  and 
resources  to  perform  lab  assignments,  and  to  do  library  research  at  their 
own  pace.  They  also  gain  the  ability  to  interact  asynchronously,  outside  of 
the  class  with  their  classmates,  teams,  and  instructors  through  chat,  Web 
board,  or  interactive  Web  based  video  technology. 

In  Type  IV  environments,  the  Web  allows  students  to  benefit  from  the  anytime  / 
anyplace  flexibility  of  earlier  correspondence  courses,  without  having  to  sacrifice  the 
spontaneity  and  interactivity  traditionally  associated  with  synchronous  modes  of 
instruction.  This  is  where  Web-based  teaching  achieves  its  maximum  contribution  in 
eliminating  time  and  space  barriers,  while  still  achieving  interaction. 

When  the  Web  is  used  to  create  a  Type  IV  environment  education  and  human 
contact  are  available  anytime  from  anyplace.  Students  can  learn  from  home,  office,  or 
wherever  they  are,  by  accessing  Web-based  lectures,  tutorials,  materials,  and  books.  They 
can  complete  and  submit  Web-based  assignments,  exercises,  and  research  interacting  in 
Web-based  forums  and  taking  Web-based  quizzes  and  exams. 

A  few  words  of  caution  are  needed  at  this  point.  Although  the  Web  can  support 
and  even  replicate  all  four  types  of  teaching  environments  this  does  not  necessarily  mean 
that  it  will.  For  example,  if  a  traditional  university  encourages  or  forces  all  instructors  to 
use  the  Web  to  support  a  Type  I  teaching  environment,  the  quality  of  instruction  may 
suffer.  Instructors  who  are  technologically  inclined  may  become  so  enamored  of  the 
technology  that  they  concentrate  on  form  rather  than  content  in  their  instruction. 
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3,  Web-based  Teaching. 

The  information  interaction  capabilities  of  the  Web  have  led  to  the  development 
of  exclusively  Web-based  courses  where  all  or  almost  all  teaching  takes  place  on  the  Web 
with  little  or  no  face-to-face  interaction.  This  mode  has  several  variations  depending  on 
decisions  made  in  the  following  areas. 

First,  course  development: 

1 .  The  faculty  member  who  will  teach  may  develop  the  Web  course. 

2.  Another  faculty  member  may  develop  the  course  and  then  supervises  the 
teaching  faculty. 

3.  A  team  of  instructional  and  Web  specialists  either  in-house  or  by  contract 
with  an  external  company  may  develop  the  course. 

Secondly,  places  of  course  delivery: 

1.  The  course  can  be  entirely  Web-based  with  faculty  and  students  never 
meeting  face-to-face  and  with  students  in  places  all  around  the  world. 

2.  It  can  be  taught  mostly  in  the  Web  with  a  few  face-to-face  interactions 
required  usually  in  the  beginning  and  at  the  end  . 

3.  The  course  can  be  taught  in  a  mixed  mode  with  some  students  in  the 
classroom,  supported  with  Web  information  interaction  and  others  taking 
the  course  entirely  on  the  Web. 

Thirdly,  timing  of  course  delivery: 

1 .  The  course  can  be  structured  so  that  there  are  time  limits  for  students  to 
complete  the  units  or  module.  For  example  the  course  starts  and  ends  at  a 
certain  date  and  the  course  or  units  or  topics  are  taken  in  locksteps  by  a 
cohort  of  students  within  a  certain  time. 

2.  Or  the  course  can  be  taken  without  time  limits.  For  example,  the  students 
are  free  to  progress  at  their  own  pace  through  the  materials  indefinitely. 
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Fourly,  level  of  interaetion; 


1.  The  eourse  ean  plaee  a  varying  degree  of  emphasis  on  using  the  Web  for 
the  transmission  of  information  for  interaetion  between  students  and 
faeulty.  It  is  always  possible  to  use  the  Web  as  a  part  of  rather  than  a 
eomplete  Web  supported  eourse. 

2.  Or  no  emphasis  ean  be  given  leaving  students  to  deeide  for  themselves. 

4,  Issues  in  Web-based  Teaching, 

Many  teehnieal  administrative,  and  pedagogieal  issues  arise  when  traditional 
universities  use  the  Web  to  beeome  fully  Web-based  anyplaee  and  anytime  edueational 
environments.  Teehnieal  issues  involve  eonstantly  ehanging  hardware  and  software  used 
in  Web-based  teaehing.  Administrative  issues  inelude  the  logisties  of  providing  remote 
students  with  the  same  support  available  as  on  eampus.  Students  for  example  may  need 
aeeess  to  the  library,  bookstore,  advising,  registration,  and  earner  serviees,  etc. 
Pedagogical  considerations  involve  the  challenges  of  quantity  control. 


B,  WHY  EVALUATE? 

1,  What  is  DoD  position? 

Evaluation  is  about  collecting  data  to  judge  the  merit  and  worth  of  any  aspect  of  a 
program.  Recently,  the  Department  of  Defense  (DoD)  has  embarked  on  a  DoD-wide 
endeavor  to  implement  Advanced  Distributed  Learning  (ADL).  According  to  the 
Department  of  Defense  Strategic  Plan  for  Advanced  Distributed  Learning  (Office  of  the 
Under  Secretary  of  Defense  for  Personnel  and  Readiness,  April,  1999),  the  Secretary  of 
Defense’s  vision  of  training  technology  is  “to  ensure  that  DoD  personnel  have  access  to 
the  highest  quality  education  and  training  that  can  be  tailored  to  their  needs  and  delivered 
cost-effectively,  anytime  and  anywhere.” 

DoD’s  ADL  program  is  making  an  important  investment  in  the  development  of 

Internet-based  resources  that  provide  high-quality  instruction  and  performance  support 

strategies  to  include  development  of  the  network  infrastructure  needed  to  support  them. 

Right  now,  while  distance  learning  efforts  are  in  the  planning  stages,  it  is  important  to 

think  in  terms  of  the  data  needed  to  provide  convincing  evidence  of  the  merit  and  worth 
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of  distance-learning  programs.  In  other  words,  we  need  objective  evaluation  guidelines 
that  will  help  answer  some  important  questions.  The  guidelines  should  address  questions 
like,  "Is  the  instruetion  and  performanee  support  of  high  quality?”  "Do  students  find 
distanee  learning  resources  convenient,  useful,  and  tailored  to  their  individual  needs?” 
and  "can  distance  learning  resources  be  delivered  cost-effectively?"  [ref  9  p.3] 

2,  Why  Start  Early? 

Planning  for  DL  evaluation  is  important  beeause  it  is  easier  and  less  expensive  to 
make  ehanges  at  the  beginning  of  a  program  than  at  the  end.  Evaluation  can  not  be 
taeked  on  as  an  afterthought.  Evaluation  planned  from  the  start  and  kept  high  on  the 
agenda  ean  produee  useful  results  with  modest  resourees  (Draper,  1999). 

When  evaluation  is  done  early  during  the  design  and  development  stages  of  a 
product,  it  is  by  convention  called  formative  evaluation.  Eormative  evaluations 
contribute  to  an  evolutionary  approach  to  development.  Early  evaluative  judgments  of 
portions  of  a  DE  produet,  as  they  are  developed,  ean  efficiently  affect  improvements  in 
the  product,  and  evaluative  feedback  is  a  form  of  quality  control.  Judgments  of 
effeetiveness  after  implementation  of  the  produet  are  called  summative  evaluations. 
Summative  evaluations  address  the  whole  DL  produet  (e.g.,  resouree  eenter,  eourse)  and 
promote  evaluation  of  the  product  in  the  wider  eontext  in  whieh  it  is  used.  They  are  also 
used  to  eompare  alternative  solutions. 

3.  Guidelines  for  Evaluation  of  Internet-Based  Instruction. 

The  Department  of  Defense  (DoD)  has  established  the  Joint  Advanced  Distributed 
Learning  (ADL)  Co-Laboratory  to  support  the  implementation  of  DL  within  DoD.  The 
function  of  the  Joint  DL  Co-Laboratory  in  Orlando,  is  to  promote  eollaborative  and  rapid 
development  of  DL  prototypes  and  DL  system  aequisitions,  principally  among  the 
Services’  training  systems  development  eomponents.  As  part  of  the  DL  initiative,  the 
Joint  Co-Laboratory  is  to  provide  teehnieal  assistance  to  program  managers  responsible 
for  the  development  and  fielding  of  DL  systems. 

The  goal  is  to  establish  documented  guidelines  for  the  design  and  evaluation  of 
Internet-based  training  and  performance  support.  Seientifioally  validated  prineiples  for 
design  are  not  yet  available  to  support  journeyman  development  of  DL  material.  It  is 
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hoped  that  the  guidelines  will  reflect  the  needs  of  DoD  and  will  be  aligned  with  DL  goals 
and  user  requirements.  The  University  of  Central  Florida’s  Institute  is  developing  the 
guidelines  for  the  Joint  DL  Co-Laboratory  for  Simulation  and  Training  [ref  10]. 

The  preliminary  evaluation  guidelines,  which  are  the  focus  of  this  report,  do  not 
yet  comprise  a  DoD  directive,  implying  mandatory  compliance  with  “standards.”  Rather, 
they  have  been  compiled  to  provide  the  best  available  guidance  and  direction  to 
evaluation  and  improvement  efforts,  which  support  the  development  of  high  quality  DL 
products.  Adherence  to  the  guidelines  is  recommended,  but  not  required. 

4.  How  Should  We  Evaluate  Distance  Learning? 

Evaluations  can  be  conducted  in  many  different  ways  and  the  evaluation  data  can 
serve  many  purposes.  The  two  methods  of  evaluation,  the  content  and  design  checklists, 
and  the  evaluation  rating  scales,  are  needed  in  the  early  stages  of  DL  program  planning  in 
order  to  give  direction  to  developers  of  DL  resources.  The  scope  of  future  distance 
learning  evaluations  will  depend  on  the  answers  to  a  number  of  questions: 

1)  Who  is  the  evaluation  for? 

2)  What  is  the  purpose  of  the  evaluation? 

3)  What  can  be  measured,  given  the  time  and  resources  allocated? 

Organizations  with  distance  learning  programs  will  decide  the  scope  of  evaluation 
by  making  decisions  about  what  outcomes  will  provide  the  best  feedback  for  distance 
learning  improvement  and  best  determine  the  effectiveness  of  their  distance  learning 
programs.  Programs  can  begin  with  an  evaluation  of  the  design  and  content  of  a  distance 
learning  product.  Instructional  developers,  content  experts,  and  other  interested 
stakeholders  can  continually  assess  the  product,  as  it  is  being  designed  and  developed. 
Then,  when  all  or  part  of  the  product  is  developed,  users  can  evaluate  the  usability  of  the 
product  and  the  usefulness  of  the  information  it  provides.  Learning  and  performance 
outcomes  can  be  obtained  by  assessing  users  themselves,  interacting  with  the  product. 
Finally,  after  distance  learning  implementation,  its  use  and  effectiveness  within  the 
organizational  context  can  be  evaluated.  As  the  evaluation  effort  is  expanded,  the  expense 
and  complexity  of  measures  increases.  Many  organizations  are  in  the  relatively  early 


11 


stages  of  the  distance  learning  planning-design-development-implementation  process, 
making  this  a  good  time  for  them  to  plan  how  distance  learning  will  be  evaluated 
throughout  all  of  the  stages.  In  the  best  of  all  worlds,  as  distance  learning  resource  centers 
and  other  distance  learning  systems  are  designed,  formative  evaluations  will  contribute  to 
highly  evolved,  high  quality  products.  Then,  as  distance  learning  products  are 
implemented,  they  will  be  subjected  to  summative  evaluations  according  to  an 
organization’s  distance  learning  plan. 
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III.  HOW  TO  DESIGN  AN  EFFECTIVE  WEB-BASED  COURSE 


A,  INSTRUCTIONAL  DESIGN  AND  USABILITY  PRINCIPLES. 

These  instruction  design  principles  (Stefanyshyn,  2001)  are  basic  guidelines  for 
instructors  on  how  to  construct  a  Web-based  instruction  course.  We  believe  that  to  have 
an  effective  online  course,  four  aspects  are  required: 

1.  A  Reliable  System:  Nowadays,  Internet  access  is  available  to  most  people. 
Students  who  bought  a  computer  system  within  the  last  five  years  can  use  it  to 
access  Web  sites.  The  hardware  requirements  for  viewing  Web  pages  are  met 
most  of  the  time  with  a  typical  system.  More  modem  hardware  and  software 
may  be  required  for  the  site  itself  but  that’s  outside  the  scope  of  our  research. 

2.  A  Well-Designed  Course  to  Capture  the  Student’s  Interest:  The  first 
impression  is  of  great  importance.  Users  should  find  what  they  are  looking  for 
at  ‘a  blink  of  the  eye’.  Otherwise  they  will  probably  click  their  mouse  to 
another  site.  To  avoid  this  in  Web-based  instmction,  the  instmctor  needs  to 
present  an  interesting  modern  first  page  representative  of  well-organized  and 
interesting  content. 

3.  A  Usable  System:  Regardless  of  the  content,  the  courseware  used  needs  to  be 
easy  to  learn  and  use.  The  system  should  not  cause  problems  for  the  students. 

4.  Allow  Instmctors  to  Work  from  a  Checklist:  ‘  Instmctors  comment  that  time 
and  the  lack  of  training  are  the  most  common  problems  in  putting  an  online 
course  together.  A  checklist  that  assists  instmctors  with  putting  an  online 
course  could  alleviate  the  time  and  burden  instmctors  have  with  putting  a 
course  online.  The  difficult  part  is  standardizing  a  generic  checklist  to  help 
instmctors.’  [Stefanyshyn,  2001] 

We  collected  information  from  traditional  literature  and  empirical  research  on 
instmctional  design  and  usability  principles.  We  then  divided  the  research  into  two 
groups:  Instmctional  design  principles  and  Usability  principles.  We  further  expanded 
these  two  groups  into  smaller  sections  for  better  understanding.  The  two  groups  and  its 
principles  are  [Stefanyshyn,  2001]: 
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1,  Instructional  Design  Principles 

a.  Course  Presentation. 

•  Present  the  material  in  simple  and  understandable  paragraphs:  There  is  a 
difference  to  interaction  between  being  in  a  classroom  with  an  instructor 
and  being  in  front  of  your  computer  screen  reading  a  course  online.  In  the 
classroom  the  instructor  may  correct  himself  on  the  fly,  in  contrast  to  a 
text  or  an  image  online  that  has  to  be  correct  from  the  beginning.  If  the 
students  do  not  understand  a  concept,  the  instructor  may  paraphrase  his 
words  and  explain  what  he  meant  to  say.  That  is  not  possible  for  an  online 
course. 

•  Keep  the  content  organized  and  as  consistent  as  possible:  “One  of  the 
more  important  rules  in  designing  instruction  is  to  have  consistency 
among  content,  instructional  objectives,  and  student  practice,  all  leading  to 
an  evaluation  that  matches  the  practice  on  which  students  have  been 
receiving  feedback”  (Berge  1998  p.73).  Each  paragraph  and  section  has  to 
be  clear  and  consistent.  To  ensure  that  students  read  the  content  and  were 
not  distracted,  hyperlinks  should  be  placed  at  the  end  of  content. 

•  Guide  the  students  from  one  location  to  another:  Links  and  hyperlinks 
provide  quick  access  to  updated  information  all  over  the  world.  While 
book  readers  turn  pages,  online  students  click  on  links  to  see  the  next 
page.  Those  links  have  to  be  obvious  and  simple,  and  link  to  the  correct 
page.  Otherwise,  students  may  find  themselves  lost  in  the  Web. 

b.  Web-based  Instruction  Principles. 

•  Getting  the  student  started:  The  site  must  attract  student’s  attention  and 
convince  him  that  he  found  what  he  was  looking  for. 

•  Presenting  the  subject  material:  Keeping  the  student  interested  in  the 
course  is  very  difficult.  In  the  classroom,  the  instructor  may  notice  that 
students  get  bored,  while  he  cannot  do  so  in  a  Web  course.  The  amount  of 
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reading  and  assignments  should  not  overload  students,  but  should  be  given 
as  modules. 

•  Assessing  the  subjeet  material:  One  way  to  measure  progress  is  through 
quizzes  and  assignments.  Instructors  should  consider  that,  “to  improve 
student  learning  performance,  the  test  should  match  the  kind  of 
information  that  was  learned  and  the  given  test  should  match  the  expected 
test”  (Najjar.  1999  p.  319). 

•  Assessing  the  course:  Students  can  give  the  best  feedback,  because  the 
course  is  targeting  them.  If  the  students  are  not  satisfied  with  the  course, 
there  will  be  no  interest  to  join  another  course.  Their  comments  are  vital  to 
improving  the  quality  of  a  course. 

•  Assessing  the  usability  of  the  courseware:  The  students  evaluate  their 
online  experience  at  the  end  of  the  course  to  determine  the  ease  of  access 
to  the  program,  delivery  of  the  material,  and  support  (mainly  technical). 
(Hayes.  2000  p.  132). 

2.  Usability  Principles  (relating  to  Instructional  Design), 

•  Ease  of  learning:  Students  want  to  learn  the  course  material  and  not  the 
application  that  delivers  it.  The  less  effort  they  put  to  learn  the  interface, 
the  quicker  and  easier  they  will  concentrate  their  attention  to  the  course. 
We  must  consider  that  users  dislike  reading  help  manuals  and  pages.  Only 
when  they  become  confused  or  lost  when  completing  a  task,  they  consult 
“Help”.  The  courseware  needs  to  be  simple  and  flexible  in  order  for  the 
students  to  learn  the  application  and  for  the  instructor  to  design  the  course. 

•  Ease  of  use:  Tf  users  have  to  continually  struggle  with  a  system  while 
learning,  users  may  develop  a  negative  attitude  towards  distance  learning, 
something  that  should  be  averted  if  ease  of  use  is  taken  into  consideration. 
Students  should  be  able  to  use  the  material  with  ease  or  very  little 
difficulty  regardless  of  user  ability. ’(Stefanyshyn  2001). 
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IV.  OUR  EXPERIMENT 


A,  SELECTING  THE  APPROPRIATE  GROUP  OF  STUDENTS 

All  our  theoretical  arguments  are  not  enough  to  justify  the  use  of  the  Web  for 
training.  We  need  to  compare  the  results  from  a  course  taught  in  a  classroom  (traditional) 
and  the  same  course  taught  online  (Web-based  training),  to  see  how  students  will  perform 
in  each  case.  Before  we  proceed  to  the  course,  and  in  order  to  have  interpretable  results, 
we  used  a  checklist  as  stated  in  Design  consideration  for  distance-learning  evaluations 
[Matthew  Champagne  and  Robert  Wisher,  p.277],  and  requires  the  following; 

•  The  existence  of  a  comparison  group. 

•  Randomly  assign  students  to  groups. 

•  Groups  comparable  in  terms  of  age,  education,  experience,  and  other  relevant 
predictors  of  outcome. 

•  Plan  before  the  training  begins. 

•  Give  a  pre-test  of  relevant  knowledge  and  skills. 

•  Administer  a  valid  or  reliable  post-test. 

This  study  is  an  evaluation  for  the  Hellenic  Navy.  Therefore,  we  have  selected  to 
bring  online  a  current  military  course.  For  the  purposes  of  our  study,  we  have  selected 
junior  officers  of  the  Hellenic  Navy,  currently  students  at  Naval  Postgraduate  School,  to 
be  our  students  and  e-students.  They  were  split  into  two  groups,  one  of  them  would  do 
the  course  in  the  classroom,  and  the  other  group  would  take  the  class  online.  The  same 
course  material  was  used  for  both  classes,  which  took  place  the  same  period.  The  same 
instructor  would  design  the  Web  course  and  teach  the  classroom  course.  In  this  way,  the 
results  would  be  interpretable. 


B,  HOW  THE  DESIGN  WAS  IMPLEMENTED 

We  have  briefly  discussed  the  design  principles  in  chapter  three.  That  was  the 
theoretical  approach  on  the  subject.  Inexperienced  programmers  tend  to  spend  no  time 
designing  their  program’s  structure  and  start  writing  code.  We  spend  several  days  in 
order  to  combine  all  the  design  principles  and  to  implement  them. 
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1,  Implementing  Instructional  Design  Principles 

•  Present  the  material  in  simple  and  understandable  paragraphs:  The  course 
was  split  into  three  lessons.  Each  lessons  was  consisting  of  four  to  five 
pages  of  text  and  tables.  Reading  a  lesson  would  take  from  fifteen  to 
twenty-five  minutes.  Each  quiz  would  take  ten  to  fifteen  minutes. 

•  Keep  the  content  organized  and  as  consistent  as  possible:  Only  content 
related  information  was  presented.  Hyperlinks  were  placed  at  the  end  of 
the  lesson  for  further  reference.  Each  lesson  was  followed  by  a  review 
summary  that  pointed  out  the  key  concepts  of  the  lesson. 

•  Guide  the  students  from  one  location  to  another:  Next  and  Previous  page 
buttons  were  located  on  the  top  and  button  of  each  page.  Additionally  a 
home  button  was  used  to  help  users  navigate  while  a  bar-drop  down  menu 
was  providing  link  directly  to  lessons,  quizzes,  contact,  and  help  pages. 
(See  Eigure  1) 
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Lessons  Quizes 


E-Contact  Surve 


<—  Back  I  Fwd  — >  | 


Welcome  to  the  E-Learning  introduction  class  for  Air  Controllers. 

This  is  an  Online  course  for  Navy  Personnel  working  in  Combat  Information  Centers. 

Modem  naval  warfare  involves  all  types  of  weapon  platforms,  such  as  ships,  submarines, 
airplanes,  helicopters  and  many  other.  This  online  course  is  addressed  to  personnel  who  gets  involved 
In  cooperating  with  an  air  asset,  usually  a  helicopter  or  a  marine  patrol  aircraft,  in  modern  naval 
operations. 

The  Air  Controller  (AC)  is  the  link  between  the  warship  and  the  air  unit,  either  helicopter  or  airplane. 

The  role  of  the  Air  Controller  who  Is  located  most  of  the  times  onboard  the  ship,  is  to  communicate 
with  the  air  unit,  ordered  to  assist  the  naval  unit  or  a  group  of  warships,  and  exchange  the  necessary 
information,  in  orderto  accomplish  the  mission  in  a  safe  way. 

You  need  about  3  hour  to  complete  this  course  introduction.  You  decide  when  to  get  started. 

Make  your  own  schedule.  You  can  start  right  now.  This  introduction  to  fundamental  takes  about  3 
hours.  Including  tests  and  you  have  one  week  to  complete  it. 

If  you  need  advice  on  how  to  use  this  site.  Click  here. 

If  you  need  brief  instructions  (less  than  5  minutes)  on  howto  take  this  course  on  line,  click  here. 

In  this  course,  you  will  learn  what  an  AC  is,  what  he  should  know,  and  what  he  can  do. 

AC  is  an  active  participant  in  aviation  safety.  He  directs  air  units  to  accomplish  tasks  in  anti¬ 
submarine  and  anti-surface  warfare... 

For  current  course  Announcements,  Click  here. 

All  new  announcements  for  the  course,  are  posted  to  this  link.  If  you  are  an  e-student,  check  here  for 
course  updates. 

To  view  the  course  Syllabus,  Click  here. 

Doni  get  started  without  the  syllabus. 

To  get  started,  Click  here. 


How  do  you  like  our  website  ? 

Please  send  any  edits  &  comments  to  our  maitto  gpstavri@nps  navy.mil  com 

Copyright  2001  -  Thesis  tesearoh  •  All  rights  reserved 


The  Home  Page  of  the  Web-based  Course  Site. 
18 


Eigure  1 


•  Getting  the  student  started;  The  home  page  (Figure  1)  was  the  most 
important  page  of  the  site,  because  we  wanted  to  get  the  student’s 
attention  and  give  him  ‘the  big  picture’.  A  welcome  message  was  followed 
by  a  brief  description  of  the  site’s  content.  After  that,  we  informed  the 
students  that  they  would  need  to  devote  approximately  three  hours  in  order 
to  complete  the  course  and  the  surveys.  That  was  important  because 
previous  usability  tests  indicated  that,  users  were  paying  more  attention  on 
how  much  time  they  had  to  spend  to  complete  the  course  and  the 
description  of  the  course  than  anything  else.  Special  care  has  been  taken 
for  new  users  who  have  never  before  had  the  experience  of  a  Web-based 
training.  The  first  link  in  the  homepage  was  a  help  page  on  how  they 
would  navigate  the  site,  what  they  were  supposed  to  do,  and  links  to 
frequently  asked  questions  (Figure  2). 


Lessons  Quizes  E-Contact  Survey  Hell 

SEARCH  THIS  SITE  FOR 


Please  follow  this  link  Frequently  Asked  Questions 


GO  TO  HOMEPAGE 


a  lesson 
Take  a  Quiz 

Chat 
online 

Syiiabus 
Ask  us  ?  ?  ? 
Survey 
F.A.Q. 


<-Back  I  Fwd  ->  | 

This  online  course  has  3  lessons,  of  15-20  minutes  each  one. 

Each  lesson  is  followed  by  a  quiz  .  5  minutes  long. 

To  navigate  use  the  main  menu  on  top,  or.  click  on  L  cp  to  homepage  ]  ggj 
started  again. 

You  may  use  a  chat  program,  the  first  time  you  use  it .  it  takes  4-5  minutes  to 
download  but  after  that  it  starts  almost  instantly. 

If  you  have  more  questions.  Please  follow  this  link  Frequently  Asked  Questions 


Send  email 


Updated:  Monday,  1 3  August 
2001  05:01  PM 


How  do  you  like  our  website? 

Please  ser)d  any  edits  &  commer)ts  to  our  mailto:  gpstavri@nps.navy-i^il-Com 


®  Copyright  2001  •  Thesis  research  •  All  rights  reserved 


Figure  2  The  Navigate  Help  Page  for  New  Users 
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^  ^  a 


GO  TO  HOMEPAGE 


Take  a  lesson 
Take  a  Quiz 


Syllabus 
Ask  us  ?  ?  ? 
Survey 
FAQ. 


Send  email^^S 

►  U  Print  page 


Updated:  Monday.  1 3  August 
2001  05:01  PM 


Lesson  1  (  Notes  on  Terms  and  definitions  ) 

Two  waycoinmuiiicjtion  titid  a  qualified  coiitiollei  aie  apieieqiiisite  for  the  control 
methods  mentioned  before. 

When  controlling  aircraft  with  the  control  methods  CLOSE-POSITIVE,  CLOSE-ADVISORY, 
LOOSE-POSITIVE  and  LOOSE-ADVISORY  the  aircraft  must  be  under  full  radar 
surveillance.  Full  radar  surveillance  means  such  a  radar  coverage  that  air  and  surface 
contacts  can  be  detected  In  time  to  avoid  separation  conflicts  with  the  controlled  aircraft. 

There  are  2  aspects  of  tactical  control  of  aircraft,  the  mission  (CLOSE.  LOOSE. 
BROADCAST)  and  the  safety  (POSITIVE.  ADVISORY)  Mission  control  types  are  normally 
associated  with  Safety  control  types:  Therefore,  the  usual  type  of  controls  are  (no  other 
codeword  is  required): 

1 .  CLOSE-POSITIVE. 

2.  CLOSE-ADVISORY. 

3.  LOOSE-POSITIVE  and 
4  LOOSE-ADVISORY 

For  reasons  of  security,  controllers  should  not  be  required  to  carry  out  control  duties  for 
long  unbroken  periods,  as  they  would  do  that  of  aircrew. 

Factors  and  conditions  determining  types  of  operations  and  control: 

1 .  Operational  considerations  (range,  communications  etc) 

2.  In  flight  conditions 

3.  Tactical  situation 

4.  Weather 

5.  Mutual  interference 

6.  Capabilities  of  Air  Control  Units 

7.  Serviceability  of  aircraft  involved 

8.  Serviceabllityofships  equipment 


Next  page  — > 


How  do  you  (ike  our  website? 

Pieese  send  any  edits  &  comments  to  our  mailto:  gpstavri@nps.navy.mil.com 


rrinhf  TODI  -  ThA«k  tA'kAAtfth  -  All  rinhK  rA<AntAi1 


Figure  3  Lesson  1  Page  3 


•  Presenting  the  subjeet  material;  The  elassroom  eourse  was  seheduled  as  all 
similar  eourses  at  NPS,  but  only  for  three  sessions  of  45  minutes  eaeh 
(Monday  through  Thursday)  and  an  in  elass  quiz  the  following  Friday.  At 
the  same  time,  students  went  online  a  given  a  week,  from  Wednesday  to 
the  following  Tuesday,  any  time  they  wanted.  The  only  restrietion  was 
that  the  quiz  had  to  be  submitted  online  by  Tuesday  noon  at  the  latest. 
Students  had  the  option  to  print  out  all  the  content  in  a  printer  friendly 
form,  without  menu  bars  and  buttons.  A  print  button  was  available,  in  case 
they  wanted  to  print  the  text  of  one  or  more  pages  of  the  course  to  read  it 
off  the  computer  screen.  As  soon  as  a  lesson  was  completed  (Figure  3),  a 
review  page  was  available  to  give  a  summary  of  what  was  taught  so  far. 
At  this  point,  students  were  asked  if  they  wanted  to  proceed  to  the  quiz  or 
take  a  pause  and  come  back  later.  Thus,  they  could  prepare  themselves  for 
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the  test,  and  take  it  when  they  would  feel  ready.  Of  course,  they  had  a  time 
limit  of  one  week  to  complete  the  course. 


•  Assessing  the  subject  material;  Both  classes  would  be  tested  at  the  end  of 
the  course,  using  the  same  quizzes.  Because  of  the  nature  of  the  Web-base 
training,  quizzes  had  to  be  on  a  multiple  choice  or  true/false  statement 
(Figure  4).  It  was  vital  on  the  success  of  the  learning  process,  to  give  e- 
students  instant  feedback  on  the  correctness  of  their  answers  and  their 
final  score,  because  then  they  could  proceed  to  the  next  lesson  without 
delays.  As  soon  as  the  quiz  was  submitted,  the  score  and  the  correct 
answers  were  displayed  on  the  screen  (Figure  5).  Students  could  repeat  a 
quiz,  but  for  our  survey  only  the  results  from  the  first  quiz  were 
considered.  All  answers  were  stored  in  our  database  for  future  reference. 


a  4}  a  Si  j 


a, 


Lessons  Quizes  E-Contact  Survey  Hel 


Fwd->  I 


This  is  Quiz  1. 

1.  The  Air  Controller  (AC)  is  relaying  orders  from  a  ship  to : 

a.  Another  ship 

b.  A  Helicopter 

c.  Fa  shore  base 

d.  F  All  the  above 

2.  The  grade  of  an  air  controller  is  significant  to  his  experience.  The  most  experience  is 

a.  F  Grade  A 

b.  F  Grade  B 

c.  F  Grade  C 

d.  F  Grade  D 

3.  An  AC  must  have  a  thorough  knowledge  of : 

a.  F  Helicopter  emergency 

b.  F  Air  control  terms  and  definitions. 

c.  F  A  thorough  knowledge  of  communication  procedures 

d.  F  All  the  above 

4  An  AC  must  be  qualified  before  he  assumes  any  type  of  control  method  for  a  helicopter.. 

a.  F  True 

b.  F  False 

5.  The  CLOSE  and  LOOSE  control  method  refer  to  the : 

a.  F  Mission  ofthe  airunit 

b.  F  Safety  ofthe  airunit 

6.  The  POSITIVE  and  ADVISORY  control  method  refer  to  the : 


Figure  4  Quiz  1 
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Take  a  lesson 
Take  a  Quiz 


Syllabus 
Ask  us  ?  ?  ? 
Survey 
FAQ. 

Send  email 

^ J  Print  page 


Updated;  Saturday.  21  July 
2001  12;11  AM 


E-Contact 

Survey 

■ 

Survev  1 
Survey  2 
Results 

Comment 

View  your  score 


Thank  you.  I 
You  can  review  the  correct  answers  for  Quiz  1. 

1.  The  Ail  Controller  (AC)  is  lelayiiig  ordeisfioin  ashipto: 

a.  ^  Another  ship 

b.  ^  A  Helicopter 

c.  ^  Ashore  base 

d-  ^  All  the  above 

2.  The  grade  of  an  aii  coiitioHei  is  significant  to  his  experience.  The  most  experience  is 

a.  Grade  A 

b.  Grade  B 

c.  Grade  C 

d.  ^  Grade  D 

3.  An  AC  must  hove  a  thorough  knowledge  of : 

a.  ^  Helicopter  emergency 

b-  ^  Air  control  terms  and  definitions. 

c.  Athorough  knowledge  of  communication  procedures 

d.  (•  All  the  above 

4  An  AC  must  be  qualified  before  he  assumes  aiiytype  of  contiol  methodfor  a  helicopter.. 

a.  ^  True 

b.  ^  False 

5.  The  CLOSE  and  LOOSE  coirti  ol  method  refer  to  the : 


a.  f*  Mission  ofthe  air  unit 

b.  Safety  of  the  air  unit 


Figure  5  Correct  Answer  /  Score  Feedback  Page 


•  Assessing  the  course  and  the  usability  of  the  courseware;  Participants 
answered  an  online  questionnaire,  in  order  to  measure  the  effectiveness  of 
the  Web-based  course,  both  on  design  and  content.  We  preferred  the  Web- 
based  surveys  for  various  reasons.  Thanks  to  standard  browsers,  little  or 
no  user  training  was  required.  The  participants  would  feel  more 
comfortable  to  answer  anonymously,  rather  than  face-to-face.  The  results 
would  be  interpreted  statistically.  Participants  chose  when  to  answer  the 
questions.  To  speed  up  that  process,  the  questionnaires  were  placed  just 
after  the  online  quiz  tests.  The  results  were  manipulated  easier  and  faster 
through  our  Web  database.  And  the  results  were  more  reliable  compared 
to  those  of  off-line  methods. 
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2,  Implementing  Usability  Principles, 


•  Easy  of  use:  As  far  as  easy  of  use,  a  standard  Web  browser  was  sufficient 
in  order  to  access  the  web  site.  The  interface  has  been  tested  through 
usability  tests  before  it  was  available  online.  So  students  had  to  put  little  or 
no  effort  to  learn  the  environment. 


-il, 


Lessons  Quizes  E-Contact  Surve 


Question  Submit  Form 


Take  a  lesson 


Take  a  Quiz 
m”  JChat 

’’^online 
Syllabus 
Ask  us  ?  ?  ? 


Please  fill  In  the  following  form: 

From;  j 

To:  gpstavri@nps.navy.mil 

Your  _ 

Name  | 

(optional) 

Your  I 

question;  I 

Submit  I  Reset  | 

My  Emaii  Address;  gpstavri@nps.navy.mil 
Last  Updated:  08/1 3/2001 


Must  be  a  vaiid  email  address  like  you@host.com 


Updated:  Monday,  1 3  August 
2001  05:01  PM 


How  do  you  iike  our  website? 

Please  send  any  edits  &  comments  to  our  mailto:  gpstavri@nps.navy.mil.com 


z] 


Figure  6  Question  Submit  Form 


•  Ease  of  learning  was  achieved  as  follows:  In  a  traditional  classroom, 
students  may  ask  questions  directly  to  the  instructor.  In  a  Web-based 
course,  this  interaction  can  be  done  in  three  different  ways.  A  student  can 
send  e-mail  to  the  instructor,  he  may  post  a  question,  or  he  may  join  a  chat 
room  and  participate  in  an  online  discussion.  All  these  options  were 
available  to  our  students.  Students  could  send  their  questions,  even 

without  an  email  program  (Figure  6).  As  an  option,  a  chat  server  (a 
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program  that  allows  users  to  connect  and  chat  online)  was  running  to  our 
site,  and  students  could  open  a  chat  client  and  join  the  course  discussion 
room.  (Figure  7). 


Figure  7  Chat  client  and  discussion  room 

C.  SOFTWARE  USED  TO  DEVELOP  THE  SITE 

The  platform  that  hosted  our  site  was  a  Windows  2000  Server,  running  Internet 
Information  Server  Version  5  (IIS  5.0).  Microsoft  Front  Page  2000  was  our  Web  page 
design  tool  for  .html  and  .asp  as  well.  Microsoft  Access  2000  was  used  to  create  the 
database  that  kept  all  our  data,  both  scores  and  survey.  The  site  was  published  at  the 
following  Web  address  http://www.movesinstitute.org/elearning,  on  July  14*^  2001,  and 
is  scheduled  to  go  offline  by  October  C*  2001.  A  freely  distributed  chat  server  and  client 
were  used  to  host  our  online  chat  rooms  [Hyperchat  suite  version  1.6  copyright  1998- 
2001  FH  software,  http://fhsoftware.tsx.org.  May  2001]. 
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D,  PROBLEMS  ENCOUNTERED  -  SOLUTIONS 

1.  Web-based  Tests. 

There  are  several  advantages  in  Web-based  tests  over  the  traditional  paper  and 
peneil  tests.  Numerous  researchers  (Bugbee  1996,  Lloyd  1996,  Glowacki  1995)  have 
indicated  their  advantages  as  the  following: 

They  have  enhanced  control  in  presenting  item  displays  and  greater 
standardization  of  test  administration. 

•  They  offer  improved  test  security. 

•  They  enrich  display  information. 

•  They  provide  equivalent  scores  with  reduced  testing  times. 

•  They  reduce  measurement  error. 

•  They  improve  the  coding  of  responses. 

•  They  can  be  automated  for  individual  administered  tests. 

•  They  can  obtain  record  at  a  central  site. 

•  They  have  immediate  test  scoring  and  feedback. 

Furthermore,  they  reduce  scoring  time  by  70%  to  80%  and  they  provide  a  simple 
mechanism  for  storing  and  retrieving  valuable  information.  Lee  (1986)  shows  that  past 
computer  experience  significantly  affects  performance  on  Web-based  tests.  As  this  type 
of  test  may  discriminate  against  those  who  have  no  experience  in  computer  usage, 
administrators  need  to  take  into  account  the  student’s  previous  knowledge  and  experience 
in  information  technology  (Heywood  1989). 

2.  Student  Identification. 

Validation  of  the  user  and  the  authenticity  of  the  submission  of  a  test  are 
challenges  in  the  Web  environment.  Open  Web  access  allows  anyone  to  sit  for  a  test  or  to 
complete  a  written  report  on  behalf  of  another.  Although  password  access  has  been 
recommended,  it  would  be  useful  to  look  beyond  more  than  one  form  of  assessment. 
Consistencies  between  synchronous  and  asynchronous  modes  of  course  participation 
could  be  reviewed  and  tracked  for  further  insights.  Short  and  frequent  impromptu  dialogs 
or  interventions  in  the  understanding  of  course  material  may  be  useful.  An  alternative, 
facility  like  a  computer  room  with  some  form  of  supervision  would  help  to  reduce  the 
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risk  of  cheating  in  a  test.  It  must  be  admitted  that  authentieation  over  the  Web  is  still  an 
issue.  In  our  study,  sinee  there  would  be  no  personal  benefit  or  detriment  for  the 
partieipants  who  were  users  e-student  1  through  10,  no  further  eonsideration  was  taken  on 
this  matter. 

E,  QUIZ  RESULTS 

All  students,  both  in  the  elassroom  and  online,  completed  the  course  on  time. 
They  have  been  tested,  and  their  seores  are  shown  in  the  following  table. 

Web-Based  Training.  In  Classroom  Training 


Name 

Score 

Student  1 

90 

Student  2 

100 

Student  3 

80 

Student  4 

90 

Student  5 

100 

Student  6 

90 

Student  7 

100 

Student  8 

80 

Student  9 

100 

Student  10 

70 

Average 

90.0 

Std  Deviation 

10.54 

Name 

Seore 

E-Student  1 

100 

E-Student  2 

100 

E-Student  3 

100 

E-Student  4 

60 

E-Student  5 

100 

E-Student  6 

90 

E-Student  7 

70 

E-Student  8 

100 

E-Student  9 

80 

E-Student  10 

90 

Average 

89.0 

Std  Deviation 

14.49 

Table  1  Test  Results 
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Statistically  there  is  no  differenee  between  the  performanee  of  students  in  the 
elassroom  and  online.  The  web-based  course  matched  up  well  with  the  traditional  course. 
The  mean  is  both  groups  is  close  to  each  other  and  the  deviation  is  small,  therefore  the 
seores  were  coneentrated  near  the  mean  score.  In  other  words,  the  online  course  matched 
up  well  to  the  classroom  course.  Because  of  the  nature  of  multiple-ehoiee  questions, 
some  students  may  have  just  reeognized  the  correet  answers  in  both  elasses.  Even  if  this 
is  the  case,  they  had  the  same  chances,  either  in  class  or  online. 


F.  SURVEY  RESULTS 

Besides  the  scores,  there  are  some  interesting  results  from  the  surveys,  which  will 
be  explained  shortly.  Our  survey  had  two  parts,  design  evaluation  and  eontent  evaluation. 
Survey  questions  and  answers  are  shown  in  Tables  1  and  2  respeetively. 

1.  Survey  One  (Design  principles) 

The  purpose  of  the  first  survey  was  to  evaluate  the  design  of  the  Web-based 
course.  Students,  through  their  answers  indieated  that  the  Web  site  was  presented  online 
in  the  form  we  initially  intended.  First  of  all,  users  had  no  problem  with  the  eomputer  and 
the  browsers.  Pages  were  loaded  fast  over  usual  modem  connections.  Navigation  through 
pages  was  very  simple  and  students  eould  “walk”  through  the  various  pages  without 
problems  using  multiple  navigation  buttons.  The  following  table  shows  the  questions 
asked,  and  the  positive  or  negatives  answers  submitted  by  the  partieipants. 


Yes 

No 

1 

Are  you  familiar  with  using  a  eomputer? 

10 

0 

2 

Did  you  conneet  to  the  e-learning  site 
easily? 

10 

0 

3 

What  connection  did  you  use? 

Modem  9 

LAN  1 

4 

Were  most  pages  loaded  in  less  than  5 
seconds? 

9 

1 

5 

If  a  page  needs  more  than  10  seeonds  to  be 
loaded,  were  you  informed? 

5 

2 

NAS 

6 

Did  you  navigate  easily  thought  the  site? 

10 

0 

7 

Did  you  find  instruetions  on  how  to  navigate 

^--4 _ _  4-U  ^ 

9 

1 
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and  use  the  site? 

8 

If  yes,  was  it  helpful? 

10 

0 

9 

Did  you  use  the  help  option? 

3 

7 

10 

If  yes,  was  it  helpful? 

3 

0 

NA7 

11 

Were  there  Back  and  Forward  buttons? 

9 

1 

12 

Did  you  have  a  Home  button  at  every  page 
you  visited? 

9 

1 

13 

If  yes,  was  it  helpful? 

9 

0 

NA  1 

14 

Were  you  ever  lost? 

0 

10 

15 

To  navigate,  you  used  most? 

All  3 

Options 

16 

Were  the  fonts  read  easily  on  every 
page? 

9 

1 

17 

Do  you  think  that  the  hyperlinks  were  at  the 
proper  place? 

10 

0 

18 

Did  you  follow  any  link? 

8 

2 

19 

If  yes,  on  average  did  you  spend  more  than 

3  minutes  to  the  linked  page? 

5 

3 

NA2 

20 

Did  you  follow  any  links  on  the  linked 
pages? 

8 

0 

NA2 

21 

Did  you  find  a  description  of  the  course  in 
the  home  page? 

10 

0 

22 

Did  you  find  the  syllabus  easily? 

10 

0 

23 

Did  you  spend  more  that  20  minutes  to 
complete  a  lesson? 

9 

1 

24 

Did  you  spend  more  that  30  minutes  to 
complete  a  lesson? 

0 

10 

25 

What  did  you  liked  most? 

Discussed 

Separately 

26 

What  you  didn't  like? 

Discussed 

Separately 

27 

Any  suggestions? 

Discussed 

Separately 

Table  2  Survey  1  and  Answers 
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Table  3  Survey  1  Results  (Graph) 
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2,  Survey  Two(Content  principles) 

The  purpose  of  the  seeond  questionnaire  was  to  evaluate  the  content  of  the  Web- 
based  course.  The  general  impression  of  almost  all  the  students  is  that  they  liked  the 
online  course.  Most  of  them  would  prefer  courses  online. 


Yes 

No 

1 

Were  you  informed  about  what  you  would  learn 
at  that  site? 

10 

0 

2 

Did  you  surf  the  site  before  start  taking  the 
course? 

7 

3 

3 

Were  requirements  to  successfully  complete  this 
course  clear? 

9 

1 

4 

Did  you  feel  comfortable  to  ask  any  questions? 

7 

1 

NA 

2 

5 

Was  it  easy  to  ask  a  question  online? 

5 

2 

NA 

3 

6 

Did  you  ask  any  questions? 

2 

8 

7 

Did  you  answer  any  questions? 

3 

7 

8 

How  many  times  did  you  use  email  to  contact 
the  instructor? 

None;  8 

1-3 

times:  2 

9 

If  yes,  did  you  get  an  answer  in  time? 

1 

1 

8 

10 

How  many  times  did  you  use  email  to  contact 
another  student? 

None  10 

11 

If  yes,  did  you  get  an  answer  in  time? 

10 

12 

How  many  times  did  you  use  the  chat  option  to 
contact  the  instructor? 

None 

13 

If  yes,  was  the  instructor  available? 

2 

1 

14 

Was  any  student  online? 

2 

1 

15 

Do  you  think  the  pictures  at  the  site  were 
relevant  to  the  content? 

8 

2 

16 

Did  you  find  the  information  irrelevant  to  the 
subject  of  the  course? 

2 

8 

17 

Do  you  think  there  was  too  much  information  on 
the  site? 

1 

9 

30 


18 

Do  you  think  there  was  a  logieal  transition  from 
one  page  to  the  next  one? 

10 

0 

19 

Do  you  think  that  information  was  up  to  date? 

10 

0 

20 

Did  you  print  any  page  to  read  off  line? 

2 

8 

21 

If  yes,  was  it  helpful? 

2 

0 

NA 

8 

22 

Did  you  take  the  quizzes  within  1  hour  after 
reading  the  lesson? 

8 

2 

23 

Do  you  think  that  the  questions  were  within  the 
eontent? 

10 

0 

24 

Did  you  find  the  eourse  interesting? 

9 

NA 

1 

25 

If  you  eould  take  the  eourse  in  elass  or  online, 
what  would  you  choose? 

8  Online 

1  Class 

room 

NA 

1 

26 

Would  you  suggested  to  other  students  to  take 
this  course? 

8 

0 

2 

Table  4  Survey  2  and  Answers. 
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Table  5  Survey  2  Results  (Graph) 
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G.  FEEDBACK  FROM  BOTH  SURVEYS. 

Besides  Yes  and  No  answers,  the  participants  were  requested  to  make  free 
comments.  Participants  were  asked  to  comment  what  they  like,  what  they  didn’t  like  and 
suggest  improvements.  It  was  very  encouraging  to  find  that  participants  submitted  many 
comments  and  suggestions  that  enriched  our  survey. 

1.  Positive  Comments 

Most  of  the  comments  were  very  supportive  of  this  effort.  They  were  impressed 
by  the  organized  course,  the  navigation  options  (a  menu  on  top  of  page,  a  side  bar,  and 
embedded  buttons  inside  the  pages  which  guided  them  through  the  course),  and  the  up-to- 
date  information.  Almost  all  found  the  course  interesting,  and  admitted  that  they  would 
suggest  something  similar  to  their  friends.  The  lack  of  negative  comments  on  design  and 
presentation  of  the  site  was  definitely  a  positive  result  of  our  effort. 

2.  Negative  Comments 

The  only  negative  comment  that  was  made  was  that  it  was  not  easy  to  get  online 
answers  in  real-time.  It  is  true  that  Web-based  courses  have  an  asynchronous  property. 
The  instructor  can’t  make  himself  available  online  24  hour  per  day,  but  he  can  check  his 
email  regularly,  or  set  up  a  schedule  to  be  online  for  chat  time.  As  I  was  trying  to  find  the 
reasons  behind  this  comment,  I  found  that  the  participant  had  visited  the  site  at  9.30  pm. 
Maybe  we  should  place  a  type  of  notification,  that  would  set  off  an  alarm  whenever  an  e- 
student  visited  the  site  and  notified  him  that  someone  was  available  to  answer  questions. 

3.  Suggestions 

Participants  have  suggested  two  possible  areas  of  further  concern.  The  first  was 
that  they  would  like  to  allow  access  only  to  authorized  users.  The  course  was 
intentionally  open  to  the  public.  Our  goal  was  not  to  test  security  or  restrict  access.  As 
three  participants  (out  of  ten)  told  me  after  the  survey  (off  the  record),  their  spouses  had 
visited  the  Website.  A  password-protected  site  would  have  prevented  those  unexpected 
but  pleasant  visits. 

The  second  concern  was  the  authentication  of  the  identification  of  the  student. 
Plagiarism  is  a  concern  for  all  Web-based  tests.  As  described  previously,  participant  had 
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id’s  as  student  1,  2,..  ,  etc  and  no  further  benefit  would  be  derived  for  the  participant. 
Thus,  there  was  no  reason  for  a  participant  to  take  a  quiz  for  someone  else.  Nevertheless, 
solutions  to  this  problem  are  a  potential  subject  for  further  study. 

A  point  of  interest  was  that  all  visitors  were  using  Internet  Explores  version  5  as 
browser.  Even  though  the  site  was  constructed  to  serve  any  kind  of  browser,  designers 
should  consider  the  fact  that  no  user  preferred  a  different  browser. 
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V.  CONCLUSIONS 


A,  SUMMARY  OF  RESEARCH  RESULTS 

As  a  result  of  this  research,  we  saw  that  statistically  there  is  no  difference  between 
the  performance  of  students  in  the  classroom  and  online.  The  web-based  course  matched 
up  well  with  the  traditional  course.  The  use  of  specific  design  principles  to  build  the  Web 
course,  contributed  to  eliminating  problems  related  to  interface. 

We  witness  comments  very  supportive  of  this  effort.  Hellenic  Navy  officers 
indicated  that  military  personnel  would  welcome  the  development  of  distance  learning 
program  within  the  Hellenic  Navy.  For  the  development  of  the  site  we  used  commercial 
of  the  shelf  (CMOS)  software  that  is  available  at  low  price  or  freely  distributed. 

A  point  of  interest  is  that  we  spent  more  time  and  put  more  effort  to  develop  the 
course  online  than  to  teach  it  in  the  classroom.  Even  if  the  course  material  was  the  same, 
the  instructor  had  to  organize  it  in  a  different  way  in  order  to  present  it  through  the  Web. 
The  author  was  instructor  of  the  same  course  five  times  in  the  past  in  a  classroom,  but  it 
was  the  first  time  he  used  the  Web  to  reach  students,  or  if  you  prefer  students  used  it  to 
reach  him.  The  technical  assistance  of  a  computer  expert  was  not  necessary  because  the 
author  had  a  strong  background  on  Web  site  developing  and  had  completed  two  courses 
related  to  Web  publishing.  We  believe  that  a  Web  design  team  must  be  available  to 
instructors  in  order  to  optimize  the  course  developing  time,  even  for  a  single  course.  It 
would  save  time  and  would  encourage  instructors  to  use  this  modem  technology  for 
teaching.  We  realized  that  the  Web  site  hosting  courses  could  be  a  virtual  meeting  point 
for  students  to  get  updated  information  after  completing  the  course. 

As  an  instmctor,  the  author  would  use  a  Web  based  course  in  conjunction  to  a 
traditional  classroom  and  as  a  place  were  former  students  could  keep  up  with  the  news. 
For  example,  instead  of  having  a  course  in  classroom  for  three  weeks,  we  could  have 
students  taking  an  online  seminar  and  then  have  them  in  classroom  for  one  week  only.  In 
that  way,  active  military  personnel  would  not  be  dispatched  from  their  units  for 
prolonged  periods.  Or  online  resources  could  be  presented  during  the  traditional  course. 
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as  an  indirect  way  to  make  students  seek  information  on  their  free  time  and  at  their  own 
initiative. 

During  this  research,  we  noticed  no  preference  in  time,  day,  or  place.  Students 
used  the  time  and  the  place  that  was  most  convenient  for  them. 


B,  FUTURE  WORK 

There  are  still  some  issues  that  need  further  investigation.  For  example,  should  a 
ship  is  underway,  how  can  we  give  the  crew  the  opportunity  to  take  a  course  online 
without  affecting  or  compromise  operations.  Another  concern  is  the  authorized  access  to 
Web  site  and  the  verification  of  the  identity  of  a  student  taking  a  course  online.  Should 
students  derive  benefits  from  participating  in  online  courses,  a  credible  system  of 
identification  has  to  be  established.  Also,  this  research  concentrated  only  in  one  class  and 
one  course,  and  not  for  the  entire  education  program  of  the  Navy.  In  order  to  implement 
the  distance-learning  program  for  the  Hellenic  Navy  to  a  wider  scale,  a  project  should  be 
initiated  in  order  to  manage  and  optimize  the  use  of  both  human  and  hardware  resources. 
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